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FIELD OF THE INVENTION 

, 5 The present invention relates to a computerized tomographic microscope for 

r3 evaluating specimens at high-resolution and a method of analyzing a specimen with use of a 

=:1]15 high-resolution three-dimensional reconstruction. 

\2 BACKGROUND OF THE INVENTION 

nj The proper evaluation of pathological specimens is of high importance to public 

1-3 health. An example is the detection and diagnosis of cancer. Often this is performed by a 

20 screening mechanism in which a cell specimen is obtained from a patient and detected under 
a microscope for abnormalities. While screening tests have decreased mortalities associated 
with cancer by preventing the development of invasive disease, it is nonetheless dependent on 
the ability of a technician viewing a specimen under a microscope to detect abnormal cells 
and structures in thick specimens or thick clusters of cells. Success depends in part on the 
25 images presented for evaluation. When the image is viewed through a conventional high- 
powered light microscope, the thickness and complexity of the specimen, the nature and 
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concentration of the stain, the microscope optics objective and the illumination source all 
influence what is seen. 

The conventional evaluation of pathological specimens using standard compound 
high-powered microscopes has a number of shortcomings. For example, the standard 
compound microscope using axial illumination is particularly ill-suited for clusters of cells. 
5 Thick or overlapping images can be distorted because of diffraction of the light and because 
of absorption by thick sections. Moreover, depth of focus is greatly reduced at high power, 
and it is difficult to visualize structures above or beyond a thin focal plane. Evaluation is 
made difficult by deficiencies in resolution, contrast, light penetration, and sharpness of 
image. As a consequence of the difficult and subjective nature of the test, there is a 

10 continuing problem with false negative readings and ambiguous readings in various 

cytopathology specimens. An example includes the Pap smear where a diagnosis is known as 
"atypical squamous cells of undetermined significance" (ASCUS). 

U.S. Pat. Application Serial No. 09/104,133 (hereinafter the ^33 application), 
discloses that oblique or multiple oblique illumination (MOI) of a specimen (or the oblique 

15 illumination equivalent achieved by tilting the specimen) , especially a cytopathology 
specimen such as cervical Pap smear, is superior to axial illumination for purposes of 
ascertaining cellular detail of diagnostic significance. Oblique or MOI illumination can be 
performed using the microscope described in U.S. Patent No. 5,345,333 (Greenberg). As 
described in the U33 application, MOI provides dramatic capabilities for penetrating thick 

20 cell clusters of cytopathology specimens and assessing individual cellular detail within 

clusters. Microscopic observation of thick cell clusters proved difficult as is well known to 
those skilled in the art using conventional axial illumination, as interpretation within the flat 
field is suboptimal. MOI provides a number of advantages, including increased intensity of 



light penetration through a wide range of focal planes within the clusters, higher resolutions, 
higher sharpness of the image, and the stereo view surprisingly and unexpectedly allowed for 
three-dimensional information to be realized. This proves invaluable for assessing cells and 
cellular detail in thick specimens or thick cell clusters. 

It would be highly advantageous if, rather than depend solely on a human observer, 
5 the results of MOI-mediated microscopic examination could be processed by a computer to 
provide 3-dimensional images amenable to further computerized analysis for cellular detail. 
To achieve this goal, however, a number of significant challenges must be met. One challenge 
results from the nature of the image the computer "sees" and the relationship of that seen 
image to the actual image in the specimen under the microscope. The image seen by the 

10 computer is the sum of minute square (or dot) individual image components, pixels, regularly 
spaced over a flat surface. Each pixel stores, in computer-readable form, the frequency and 
intensity of light that impacted that pixel. Because, however, microscopes cannot be 
perfectly focused, the pixels seen by the computer will have received light not only from the 
focal plane of interest (i.e., a "virtual" thin slice of the specimen), but also specimen 

15 components just below and just above that plane. The present invention provides a method 
for "cleaning up" such images so that they more closely represent only the image in the focal 
plane of interest. To accomphsh this goal, the computerized process uses a novel algorithm 
and analyzes a plurality of images of a single specimen plane, each image taken using a 
different source of oblique illumination. The resulting images can be stacked by the computer 

20 to provide a 3-dimensional image of the specimen, which image can either be displayed on a 
computer monitor or be further analyzed by a computer to quantitatively characterize 
components of cells in the specimen. 
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SUMMARY OF THE INVENTION 

In one general aspect, the invention is a method for evaluating a specimen comprising 
the steps of: 

(1) obtaining a plurality of images of the specimen along a plurality of planes defining 
the thickness of the specimen, in which the plurality of images comprise at least two 

5 images taken at different angles relative to each other; and 

(2) cross-correlating information obtained from the angled views to produce a single 
output image for the each one of the plurality of planes to define a plurality of 
confocal emulated images. 



10 In one embodiment, the process comprises a step (3): 

(3) stacking the plurality of confocal emulated images to produce a three-dimensional 
reconstruction of the specimen. 

It is preferred that, in the method, for all the images are obtained in a given plane and 
plane area, the locations of said plane and area relative to said specimen are the same at the 
15 time all images are obtained. 

It is also preferred that, in step (1), each image is obtained on a surface comprising a 
regular arrangement of pixels, each pixel capable of representing absorbed light or emitted 
light (fluroescence) so that information on both the frequency and intensity of the light 
absorbed or emitted is computer readable. 
20 In a preferred embodiment of the method, in step (2), the information cross-correlated 

comprises absorbed or emitted light frequency information and intensity for each pixel. Most 
preferably, the cross-correlation comprises a cross correlation of information between a 
linearly arranged sequence of pixels from each image, the sequences at least partially 




overlapping. Furthermore 

it is preferred that the cross-correlation will, for purposes of constructing a confocal emulated 
image, score as undesirable that specific image information that is displaced in one image 
relative to the other and will score as desirable that specific image information that is not 
displaced in one image relative to the other, and further for purposes of constructing a 
5 confocal emulated image will tend to retain desirable information and to eliminate 
undesirable information. 



In another general aspect, the invention is a tomographic microscope (Dlustrated in 
Figure 1 and in part in Figure 2) for evaluating a specimen, said microscope comprising: 
10 (a) a means for obtaining a plurality of images of the specimen along a plurality of 

planes defining the thickness of the specimen, in which the plurality of images 
comprise at least two images taken at different angles relative to each other; 
(b) a means for cross-correlating information obtained from the angled views of 
step (a) to produce a single output image for the each one of the plurality of 
15 , planes comprising a confocal emulated image for each one of the plurality of 

planes to define a plurality of confocal emulated images. 



The means for (a) and (b) are illustrated by Figures 1, 2, and 3, in combination with 
the computer algorithms described herein. The computer algorithms are implemented by the 
20 computer shown in Figure 1. Similarly all other embodiments of the device, means described 
therein and computer algorithms related thereto can be understood by reference to Figures 1-4 
and the algorithms described herein. 

In a preferred embodiment, the device further comprises a means for stacking the 
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plurality of confocal emulated images to produce a three-dimensional reconstruction of the 
specimen. 

In a preferred embodiment, the means for cross-correlating information implements 
the cross correlation for data obtained where all the images are obtained in a given plane and 
plane area, and the locations of said plane and area relative to said specimen are the same at 
5 the time all images are obtained. 

It is also preferred that, in the means for obtaining a plurality of images of the 
specimen, each image is obtained on a surface comprising a regular arrangement of pixels, 
each pixel capable of representing absorbed light or emitted light (fluorescence) so that 
information on both the frequency and intensity of the light absorbed or emitted is computer 
10 readable. 

In a preferred that the means for cross-correlating information implements the cross 
correlation for data obtained where the information cross-correlated comprises absorbed or 
emitted light frequency information and intensity for each pixel. 

Most preferably, the means for cross-correlation cross correlates information between 
15 a linearly arranged sequence of pixels from each image, the sequences at least partially 
overlapping. 

It is preferred that the means for cross-correlation will, for purposes of constructing a 
confocal emulated image, score as undesirable that specific image information that is 
20 displaced in one image relative to the other and will score as desirable that specific image 

information that is not displaced in one image relative to the other, and further for purposes of 
constructing a confocal emulated image will tend to retain desirable information and to 
eliminate undesirable information. 
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In another aspect the invention is electronically, magnetically, or electromagnetically 
stored data generated by a method of this invention. 



5 BRIEF DESCRIPTION OF THE DRAWINGS 

The advantages, nature and various additional features of the invention will appear 
more fully upon consideration of the illustrated embodiments now to be described in detail in 
connection with the accompanying drawings. In the drawings: 

Figure 1 is a schematic illustration of one embodiment of the computerized 
10 tomographic microscope including use of a mechanism for performing multiple-beam oblique 
illumination. 

Figure 2 is a schematic illustration of part of the microscope of Figure 1 plus a 
schematic representation of a pixel intensity pattern created using the microscope. 

Figure 3. Example of multiple mirror arrangements relative to the main axis of the 
15 optical system: Three mirror arrangement. 

Figure 4 is a block diagram showing the steps of the method of analyzing a specimen 
with use of the tomographic microscope and a high-resolution three dimensional 
reconstruction. 

Figure 5. Images of a Pap Smear sample. The left panel is the image taken at one 
20 oblique angle. The right panel is the image created from two oblique angles (separated by 
180°) by correlation analysis. The black areas represent the images created by a nuclear dye. 
Outlines of cell and their nuclei are evident in the right panel which is much sharper (in 
focus) than the image on the left. 
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It is to be understood that these drawings are for purposes of illustrating the concepts 
of the invention and, except as noted below in the description are not to scale. 
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DETAILED DESCRIPTION 

Applicants have discovered a device and method for performing high-resolution 
tomographic microscopy based on angled views, images or light paths, computerized 
confocal emulation, and high-resolution three-dimensional image reconstruction. With this 
invention, detail in cell clusters or thick specimens such as obtained from Pap smears can be 
accurately reconstructed in three-dimensions at high resolution. Also, as another example, 
neurons and dendrites could be likewise reconstructed in a specimen consisting of thick brain 
tissue (e.g., having a thickness of 30 to 40 microns). Out of focus haze is dramatically 
eliminated, and nuclear and cytoplasmic detail may be selectively obtained with use of a color 
segmentation algorithm. The invention comprises creating high-resolution animation sets of 
rotating cell cluster volumes to produce significant increases in sharpness, contrast, and 
resolution. 

Referring to the drawings. Fig. 1 schematically illustrates the computerized 
tomographic microscope including use of an optical system for multiple oblique illumination 
(MOI). Such MOI systems are described in greater detail in United States Patent No. 
5,345,333 which is incorporated herein by reference. Microscopes employing oblique 
illumination or oblique view (known as the 3-D microscope head) are commercially available 
from Edge 3-D Imaging, LLC, Philadelphia, PA. Single beam oblique illumination can also 
be achieved by rotation and tilting of the stage or of the condenser in an axially illuminated 
microscope. 

One can, for example, take a microscope created for MOI and utilize the 
camera port on the microscope to take images for the algorithm of the present invention. To 
that end, one uses one oblique angle light source at a time to create an image and then, using 
the algorithm described herein, combines images from different light sources to create the 




final image. An additional possibility is to use two polarized light sources, the light from the 
sources distinguishable based on their polarization. The polarized light from both sources can 
be used simultaneously to illuminate the specimen. 

It follows that, although the tomographic microscope works well with MOI, MOI is 
not necessary for the invention. Another traditional image capturing device may be used 
5 including a traditional axial microscope, the important consideration being that the image- 
capturing device must be capable of obtaining views of the specimen at different angles, e.g., 
by rotation or tilting of the specimen, the light paths, by altering one or more light paths used 
in the device, or with polarized light. 

The invention can be further understood by referring to Fig. 1. which figure illustrates 

10 a device 2 of this invention. In that Figure, a cytopathology specimen 10 is shown mounted 
on a slide 102. Schematically at box 104 there is illustrated a means for rotating the slide and 
specimen relative to the light beams 33, 34 of the MOI apparatus. The specimen should have 
a thickness that permits visualization of the specimen and the acquisition of images from a 
plurality of angles. The thickness of the specimen will depend upon the magnification of the 

15 optical system. For example, even a single cell can have the necessary thickness at lOOOX 

magnification under conditions of bright field or fluorescence light microscopy. However, in 
most cases the specimens will have a thickness ranging from greater than 4 to 5 microns or up 
to 100 microns or more. This same principle of tilting the view or illumination source could 
apply to electron microscopy as well. This could include both scanning or transmission 

20 electron microscopy. This will apply to any tilted or oblique images so obtained. 

In the illustrated optical system of Fig. 1, which uses multiple oblique beams, a 
plurality of beam path shifters such as mirrors 23 and 24 are disposed off the optical axis 13 
of condenser 12 to permit the system to operate with plural independent illuminating beams 
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to the condenser lens. A first light source 26 directs a light beam 27 along a first beam path 
28 that includes mirror 23. Similarly, a second light source 29 directs a second light beam 31 
along a second beam path 32 which includes mirror 24, Mirror 23 shifts the direction of 
beam path 28 to path 28a passing through condenser 12. Mirror 23 is disposed a distance 
radially away from condenser axis 13 and at an angle n relative to the incident light beam 27 
5 which produces the exit beam path 33 from condenser 12 to emerge from the edge of the lens 
at the maximum angle which includes the objective 14. (See U.S. patent 5,345,333 for an 
example of how to achieve variable n ) Similarly, mirror 24 shifts the direction of beam path 
32 to path 32a which includes the condenser 12. Mirror 24 operates in precisely the same 
way as mirror 23 to produce the desired exit beam path 34 from condenser 12. Specimen 10, 

10 which can include clusters of cells or other structures, is disposed at the intersection of the 

oblique beams 33 and 34. The beams projecting through the specimen enter the objective 14 
and the specimen can be captured by a device 42 and digitally processed as by computer 44. 

According to the invention, the apparatus obtains a plurality of images along a first 
plane of the specimen. In one embodiment, two images are obtained at angles of view taken 

15 at ISC' relative to one another (Fig. 1 and 2). These two images are cross-correlated with use 
of an algorithm, described in more detail below, to obtain a thin plane of detail common to 
both images . In another embodiment, three images are obtained at angles of view taken at 
120"^ relative to one another (See Fig. 3) . These images are cross-correlated with use of an 
algorithm, described in more detail below, to obtain a thin plane of detail common to all 

20 three images (e.g., center, right and left, or three 120^ images) and provide a confocal 

emulated image. In another embodiment, an image is obtained at the center of the specimen, 
as shown, and then the means 104 for rotating the specimen around the main optical axis (or 



11 




other means for obtaining angled views) is activated to obtain images to the right and left of 
the specimen center. 

Although use of two or three images is noted above, it should be understood that a 
greater number of images can be obtained, and even an indefinite number of angled views or 
images is possible. However, at each plane of the specimen there should be views taken from 
5 at least two different angles. It is contemplated that the best results will be achieved when 
images are obtained from 2 to 36 angles relative to the specimen at each plane. The 
computerized microscope will obtain an output (confocal emulated) image for each one of the 
planes, and the process repeated for a multiplicity of planes along the depth of the specimen, 
3 The output image for each plane is stacked to obtain a three-dimensional reconstruction of the 

;^10 specimen. The three-dimensional reconstructed image is displayed on monitor 44. This 
ij dramatically improves resolution in the third dimension or z-axis in reconstruction. 

3 The significance of the optical system and procedure illustrated in Fig. 1 can be 

=- further appreciated by referring to Fig. 2. The purpose of Fig.2 is to illustrate that the 

7: relationship between the macroscopic features of the device and the microscopic features 

;U5 within the specimen 10. Fig. 2 makes some modifications of elements of Fig 1.: The beams 
32, 34 and 58, and related components of the beam path are represented as three parallel 
arrows instead of one to emphasize the macroscopic width of the beams. The specimen 10 is 
artificially enlarged in Fig. 2, so that microscopic balls 46 and 48 of equal size can be 
schematically represented. Generally, a microscopic element of interest in a specimen will not 
20 be a perfect ball, but they are used here for illustrative purposes. 

The microscopic balls 46 and 48 are stacked along the vertical axis through which the 
beams of hght pass on their way from the condenser 12 to the objective 14. Ball 46 is in the 
focal plane 60 of interest, whereas ball 48 is slightly below the focal plane of interest. 
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Because ball 48 is slightly below the focal plane of interest, it will cause a smaller amount of 
light to impinge on the pixel-coated surface 50 than the ball 46 will. Furthermore the light 
due to ball 48 will impinge at a pixel(s) slightly displaced from that were the light due to ball 
48 impinges. 

The differences in the effects of balls 46 and 48 on the light absorbed by the pixilated 
5 surface 50 are shown at the top of Fig. 2. The intensity for the situation "R on" shows that the 
intensity of light due to the focal plane ball 46 has a greater intensity 52 (the height of the line 
52 is proportional to the intensity) than the intensity 54 due to the ball 48, Furthermore, the 
light from ball 48 impinges on surface 50 slightly to the left of where the light from ball 46 

'i impinges. (Note the scale of the horizontal axis 56 of the diagram is in pixels whereas that of 

310 the surface 50 is of the order of nun. In essence, the Intensity vs. pixel diagram provides an 
exploded representation of a small portion of the surface 50) If the left light L is on, but the 
right light R is off, then the pixel positions of the light from the balls is reversed. This 
reversal is seized upon by the computer algorithm to predict that the intensity 54 is due to an 

iJ out of focus component. Analogous reversals would be seen for a ball slightly above the focal 

45 plane ball 46. 

The instant invention also comprises a method of analyzing a specimen with use of 
the tomographic microscope and a high-resolution three-dimensional reconstruction. An 
overview of the method is illustrated in block diagram of Fig. 4. A first step of the method 
involves providing the specimen which as aforementioned is relatively thick, depending upon 

20 the magnification that is used to view the specimen. A next step, in block two, is to obtain a 
plurality of images comprising angled or tilted views along a first plane of the specimen. 
Preferably this step is performed with a MOI apparatus although other image-acquisition 
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apparatuses could be used as well, providing an angled or tilted or polarized view or 
illumination. 

A next step (block4) comprises applying a computerized confocal emulation 
algorithm to cross-correlate information obtained from the angled views to produce a single 
output image for each plane of focus of the specimen. Essentially, with the algorithm, 
information that is not common to each angled view is omitted and a plane of detail is 
provided that is conmion for all angled images. Thus, a high-resolution image, avoiding haze 
or other out-of-focus information is produced. Blocks 2 and 4 are repeated for a multiplicity 
of planes through the thickness of the specimen. The order in which the steps are performed 
is not critical. For example, a plurality of views across a first plane may be taken, a confocal 
emulated image for the first plane obtained (e.g., with an algorithm), and then a plurality of 
views taken across a second plane, a confocal emulated image obtained for the second plane, 
and so forth (e.g., following arrows A of Fig, 4). Alternatively, a plurality of angled views 
may be taken across the first, second, third, and so on planes (e.g., following arrow Bi of Fig. 
4), and then the confocal emulated image determined for each one of the focused planes after 
all the views have been imaged (arrow B2). Another approach would be to follow a 
combination of these methods, e.g., following both arrows A and B of Fig, 4, 

Once the high-resolution images are obtained for each plane of focus, the images are 
stacked (e.g., one image placed on top of another) and software applied (e.g., Voxblast 
software available from Vaytek, Inc. Fairfield, Iowa, and others) to produce a three- 
dimensional reconstruction of the specimen. The invention results in greater resolution, more 
accurate measurements, and provides superior results for automated procedures, as all the 
information input into the computer in performing the reconstruction comprises high- 
resolution information. 
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Computer Algorithm for the Creation of High Resolution Images from Serial Sets of 
Stereoscopic Images 

The objective is to enhance material in the focal plane of each section while 
suppressing or eliminating material in proportion to the extent to which it is out of focus 

The two most obvious parameters that vary with focus are lateral displacement 
between the left and right images of a stereoscopic pair and the strength of high frequency 
information associated with each visible object. Any operation that enhances high frequency 
information at the expense of low frequency information (e.g. high pass filtering, unsharp 
masking) will enhance in-focus material while suppressing out of focus material. The 
inventors have already observed that simple high-pass filtering has been very effective at 
removing large, dense out-of focus objects while enhancing small, pale in-focus objects. One 
disadvantage of high-pass filtering is that large, homogeneous in-focus objects may be 
suppressed. However the edges should still be sharp and therefore enhanced. Also, 
projections of in-focus objects should contain high frequency information within the object 
edges although some of this may be obscured by low-frequency contributions from out-of- 
focus planes in the case of thick objects. It is not common for in-focus projections of 
biological material would have homogeneous interiors. Thus high -pass filtering was 
considered for the algorithm. 

It was found that more reliable results may be obtained from the parallax associated 
with stereological pairs of images of objects at different focal levels. The closer an object is 
to the focal plane, the less the displacement of the object between members of the 
stereoscopic image pairs. The simplest way to take advantage of this is to divide the product 

15 



of the left and right image by the difference image. However, there are a number of situations 
where this will not work properly: 

• Large out-of-focus objects may still cause a large number of pixels to be of high density in 
both images. This would be caused by the overlap which is the difference between the lateral 
width and lateral displacement of the object. 

• Different small out-of focus objects may move into each other's positions resulting in the 
corresponding pixels being dense (and possibly of similar density) in both images. This is 
more problematic for the product image than for the difference image since high density 
pixels would be at the same wrong location in both images but it is less likely that the 
densities would be as similar as for the case where an in-focus object occupies the same 
pixels in both images. 

• Larger out-of-focus objects may move to cover smaller in-focus objects. This would cause 
a problem with the difference image but not with the product image. 

Correlation analvsis . Another means of analyzing object displacement is through 
correlation analysis. This approach has the advantage of being able to analyze pixels in 
relation to their neighborhoods instead of in isolation. Since displacement between images is 
1 -dimensional, 1 -dimensional correlation analysis appears to be appropriate. A sampling 
window of a certain length is used to obtain sample vectors from groups of pixels in 
corresponding locations in each of the two images. A corresponding vector pair (CVP) 
consists of two vectors, one from each of the left and right images, comprising a contiguous 
string of pixels with the same coordinates in each image. Cross-correlating these vectors 
provides as measure of the similarity between corresponding (1-D) neighborhoods in the two 
images. The result of cross-correlation is a correlation vector. 

If the CVP members are normalized (divided by their respective norms) prior to cross- 
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correlation then the highest member value in the resulting correlation vector will be 1.0 if the 
non-zero elements of the CVP members are identical and have the same configuration. In 
this case, the nonzero elements of the correlation vector v^ill also be symmetric about the 
peak value. If the nonzero elements also occupy the same pixel locations in both CVP 
vectors, the peak value will be the central element of the correlation vector. If the non-zero 
elements are displaced between the CVP members then the peak value in the correlation 
vector will be displaced in proportion to this. However, if the non-zero elements are identical 
and have the same configuration apart from the displacement then the peak value of the 
correlation vector will be 1.0 and the non-zero elements will be symmetric about this 
maximum value. If the non-zero elements differ between the CVP members and/or they have 
different configurations then the maximum value of the correlation vector will be less than 
1 .0 and will be proportional to the similarity between the configurations of the non-zero 
values. 

If the CVP members are not normalized prior to cross-correlation, then the values of 
the vector elements are reflected in the elements of the correlation vector. Hence cross- 
correlation of dissimilar vectors with large elements will yield vectors with larger elements 
than cross-correlation of identical vectors with small elements. Hence normalization is 
required to decouple similarity from intensity. Normalized correlation measures the 
similarity between the relative, rather than absolute, values of vector elements. It is therefore 
theoretically possible that high correlation could be measured between an object in one image 
and background in the other. However this would require high similarity in the configuration 
of the relative values of elements of the vector sampling the object in one image and the 
background in the other. The probability of this is low and falls with the size of the sampling 
window. Hence the sampling vectors should be normalized prior to cross-correlation. In 
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analyzing the correlation vector, displacement of the peak from the center indicates 
displacement of the object between sample images. Since this displacement indicates 
displacement of the object from the focal plane, it was considered appropriate to use only the 
center element of the correlation vector when measuring the similarity between samples. 
This allows for a large sampling window to be used without high correlation being interpreted 
5 from an out of focus blob being present in the sample from both windows. 

Correlation analysis is expected to be quite successful in identifying small objects that 
are in focus. It would be less successful with objects that are large relative to the sampling 
window since, when such objects are out of focus, the pixel values will change smoothly and 
slowly over several windows. Hence, high correlation would tend to be measured between 

10 vectors that do not correspond to the same part of the object. It is also possible, although 
much less likely, that a single small similar object may be seen at the same location in both 
CVP members but they may be different out-of-focus objects. In this case, high correlation 
will be interpreted but inappropriately. The risk of this problem may be reduced by using a 
larger sampling window and/or using 2D correlation. However, the effect of using one or 

15 both of these approaches would be to reduce the probability of this problem arising and not 
precluding it. Also, small out-of-focus objects would tend to have a similar (Gaussian) 
configuration in 2D so using 2D analysis is expected to be less preferable for dealing with 
this problem. 

Optical flow approach . Since lateral displacement is important in identifying in-focus 
20 objects it was considered appropriate to use optical flow for this purpose. The optical flow 
approach incorporates and extends the principles of correlation analysis. A 2D optical flow 
image would consist of pixels each representing the displacement vector from one image to 
the next. For example, an optical flow image from the left image to the right would consist of 
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vectors each describing the magnitude and direction of the movement of whatever is at the 
pixel's coordinates in the left image to wherever it will be located in the right image. Such 
vectors could be fully defined by a complex image (where each pixel has a real component 
and an imaginary component). The magnitude of displacement would be the hypotenuse 
(square root of the sum of the squares) of the real and imaginary component. The direction 
5 would be the arctangent of the imaginary component divided by the real component. 

However, this is more than is required for this application since movement is confined to one 
dimension and it is only the magnitude that is required to determine the distance of the object 
from the focal plane. If it is desirable to determine whether the object is above or below the 
focal plane, this information may be stored by applying a positive or negative sign 

10 (representing the ID direction) to the optical flow pixel's magnitude. However, such 

information is not germane to the present objective. Hence, all that is required for each pixel 
of the optical flow image is the magnitude of the displacement. The smaller this value, the 
more in-focus the object in that part of the original image will be. 

An optical flow image can be easily obtained using ID cross-correlation. Since 

15 movement (between the left and right image) is confined to the horizontal axis, the optical 
flow image is obtained by taking ID window samples of the left image ("left image" here 
means an image taken from lamp on the right) and cross-correlating each sample with every 
possible window sample (taken after each of a series of 1 pixel shifts of the window relative 
to the window for the left image) on the same row of the right image. Window samples in the 

20 left and right image that overlap therefore are obtained by 1 -pixel incremental sampling. 

Samples are normalized prior to cross-correlation. The sample in the right image, for which 
the middle element of the correlation vector is highest, is taken to be the matching window in 
the right image. The magnitude of the displacement (number of required pixel shifts to 
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obtain overlap) fron, *e window .n .he ,eft .mage to the ma,ch,„g window .n .he righ. image 
is .hen assigned .o .he optical flow pixel w,ft coordina.es corresponding .o .he middle 
elemen. of .he sample window in *e left image. One problem w.<h *,s ,s d.a. i. is .he 
highes. valued elemen.. ra.her U.an .he m,dd,e elemen. .ha. will maKe .he grea.es. 
con.nbu.ion .o .he cross-correlaUon vec.or. Hence *e d,sp,acemen. vec.or ma,nly reflect 
.he displacemen. o, .he h,ghes.-va,„ed elemen.. The«fcre. i. is preferable .ha. .he sampling 
window is no. so large as .o incorporare more ftan one objec. a. a .ime. 

Opiical flow is expected .o be a superior approach .o simple correlation since 
correlation could be high w.th similar objec. bu. flow «ies u, ma.h parricular ob,ec.s and 
^ord .he displacemen.. I. is also possible d,a. image flow could be ex.ended .o ge. dep.h 
information. Anodrer advanrage of optica, flow is ti,a. one can co„«,l .he dep.h o, fleld b, 
se.ting a flow magni.ude .h^shold. ObjeCs assoc,a.ed wi.h flow above .h,s .hreshold are 
removed. 

one dimensional conflation analysis is done, for example, using a correlation 

The ^ay scale pixel system is a useful one for implementing *e inven.ion. bu. any 
o.her pixel sys.em (e.g RGB) can also be used. - - 



once each indivdual plane of .he image is cons.ruc.ed. ^own algorrthms and softwa. can 
he done .o combine (s.aclc, .he images in.o a 3-dimensional image. Similarly known 
algorisms and software can be used .o ei.her projec. tirose images on a compu.er moni.or or 
calculare .he volume of entities (e.g. a cell nucleus) wi.h,n the image. Volumes of specftc 
entities can be calculated, for example, by identifying the,r boundaries by Icnown algon-hms. 
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calculating the cross-sectional area of the entity in a given plane, and integrating the areas 
over all the planes. 

To improve both viewability and/or computer generated images, the images can be 
inverted (black <-> white) to insure that the objects are brighter than the background. 
It is also helpful to median filter the optical flow magnitude field. 
5 It is also helpful to modify the left imge as follows: If a pixel has a value greater than 

a user-specified maximum value, it is set to the highest value (255 for 8-bit images). If a 
pixel has a value less than a user-specified minimum value or the corresponding pixel in the 
OFMF image is greater than the user-specified maximum, it is set to zero. In other cases, the 
^ pixel is left unchanged 

plO 

U1 General Applications 

^ The instant invention has many applications. The invention may be applied for any 

I ^ clinical analysis of a specimen having a thickness such as a cluster of cells or even a single 

i [j cell at high magnification (lOOOX). One application involves analysis of PAP smears. It can 

r:j 15 be used to obtain quantitative information on such diagnostically relevant intracellular 

information such as nuclear volume, cytoplasm volume, microtubule distribution, cell shape, 
and the color of dye-stained cytoplasm. 

The results of the computer analysis may also be used as the basis for 3-dimensional 
representation of cell and its discrete internal organelles, such as its nucleus, on a computer 
20 monitor. 

Three dimensional volumes can be calculated, for example, by identifying the 
boundary of interest and calculating the cross sectional area of individual slices (planes) of 
the volume and then integrating over all planes - all of which can be done with well known 
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algorithms. 



m 



EXAMPLES 

Optical sections of thick (10 to 40 micron) cell clusters from cervical Pap smears were 
5 captured sequentially at 400x using various oblique light paths. Images were then processed 
with a computer algorithm to produce confocal emulation. Three-dimensional 
reconstructions were then complied with the high-resolution images using Voxblast software 
(available from VayTek, Inc. of Fairfield, Iowa). Cells were color segmented, and various 
volumetric characteristics were emphasized. 
10 An illustration of the result of using the method of the invention is shown in Figure 5. 

It is understood that the embodiments described therein are merely exemplary and that 
a person skilled in the art may make variations and modifications without departing from the 
spirit and scope of the invention. All such variations and modifications are intended to be 
included within the scope of the appended claims. 
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